Learning Objectives

After completing this lesson, you’ll be able to:

• List and describe various data sources that can be used in FME.
• Explain different methods for source data management.
• Understand how to publish source data to an FME Flow repository with the Publishing Wizard.
• Understand the limitations of publishing data to an FME Flow repository.

Introduction

Nearly every FME workspace starts by reading features from a source dataset.

In some cases, that source dataset may be held in a database or stored on a web service such as Google Drive; it may even be a database running on a web service! On other occasions, the data may not be web-based at all, but stored on a file system and shared for others to access.

In general, it's easy to author a workspace for use on FME Flow because what works on FME Form can be published and run on FME Flow with minimal alteration of the source data.

However, there are additional methods that can be used to take source data management on FME Flow to the next level. These methods include:

• Managing file datasets.
  • Publishing file datasets to a repository.
  • Uploading file datasets at run time.
  • Storing file datasets on the 'Resources' file system.
• Using Database Connections for data stored in databases.
• Using Web Connections for datasets stored on a web service.

This course will cover each method in turn.

If you want to learn more about data integration in general, we recommend you browse these learning resources:

• Why Data Integration?
• What is Data Integration?
• FME Integrations Gallery

Publishing File Datasets to a Repository

First, let's look at one of the simplest methods: publishing data to a repository. When the source data for a translation is stored as files (rather than a feed or database), it is possible to publish data to an FME Flow repository along with the workspace. This data upload method is fast and simple but limits future access to the data. A better practice is to publish your data to the Resources folder separate from the workspace, but we'll walk through those steps later in the course.

Publishing Source Data

In this workspace the source dataset is MapInfo TAB:

A MapInfo TAB dataset is made up of a series of files (.tab, .dat, .id, .map). When this workspace is published, the wizard allows us to publish the data files alongside it by simply checking the box labeled Upload data files.

FME automatically selects the files to upload based on what it thinks is necessary to run the translation. If there are other files you wish to upload, or files FME selected that you don't wish to upload, the Select Files button allows you to make changes:

This dialog also allows you to change where the files are published, but for now, we'll ignore that setting and go with the default of publishing to the repository.

Once the publishing wizard is complete, those files are uploaded to FME Flow and tagged for use with this workspace.

Using Published Source Data

When a workspace published with its data is run on FME Flow, the uploaded data automatically is used as the source:

There are no other settings that need to be changed, and the workspace will run to completion using the published data as its source.